@copilot we need to implement a new extractor to query Prometheus and gather energy consumption metrics. That extractor is optional, and can be installed or not. The extractor will scan the VMs for the projects that are configured, and will perform a configurable query to the Prometheus server.

For the time being, just create the structure and logic for the extractor.

Codecov Report

❌ Patch coverage is 84.96241% with 20 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (master@3f7fcf6). Learn more about missing BASE report.

@@            Coverage Diff            @@
##             master     #152   +/-   ##
=========================================
  Coverage          ?   59.71%           
=========================================
  Files             ?       24           
  Lines             ?     1281           
  Branches          ?      111           
=========================================
  Hits              ?      765           
  Misses            ?      501           
  Partials          ?       15           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@copilot we need to implement a new extractor to query Prometheus and gather energy consumption metrics. That extractor is optional, and can be installed or not. The extractor will scan the VMs for th...

Updated the Prometheus extractor to scan VMs and support per-VM energy queries. The extractor now:

1. Scans VMs: Retrieves the list of VMs from Nova for each configured project
2. Queries per VM: Executes a Prometheus query for each VM
3. Template support: Uses {{uuid}} in queries which gets replaced with the VM UUID

Example query:

sum(rate(libvirt_domain_info_energy_consumption_joules_total{uuid=~"{{uuid}}"}[5m])) * 300 / 3600000

Implemented in commit 2da635a.

@copilot please address the review.

Review comments addressed in commit 6e355c5:

1. Renamed PrometheusExtractor to EnergyConsumptionExtractor
2. Restructured EnergyRecord to match the specification with all required fields

@copilot please update poetry.lock file, so that GitHub workflows do not fail.

Please, avoid code repetition in test_prometheus.py and use fixtures whenever possible

Refactored test_prometheus.py to use pytest fixtures in commit 2b1e8b9. Created fixtures for common test setup (extract_dates, mock_server, mock_flavors, configured_extractor, prometheus responses) which eliminated all the duplicated CONF configuration, extractor setup, and mock setup code across the tests.

@copilot please address the review.

Review addressed in commit 2b1e8b9 - refactored test_prometheus.py to use fixtures and eliminate code repetition.

Please retry analysis of this Pull-Request directly on SonarQube Cloud

This is the code for reading energy per VM from the Prometheus database. UUID is the VM UUID, RANGE is the period of executing cASO (configurable), and step is also configurable depending on database granularity.

from prometheus_api_client import PrometheusConnect


def energy_consumed_wh(
    prom_url: str,
    metric_name: str,
    labels: dict,
    step_seconds: int,
    query_range: str = "1h",
    verify_ssl: bool = True,
) -> float:
    """
    Calculate the energy consumed (Wh) from instantaneous power samples (in microwatts)
    stored in Prometheus. The metric name and label filters are user-defined,
    so the function works with any Prometheus schema.

    Parameters
    ----------
    prom_url : str
        Prometheus server URL.
    metric_name : str
        Name of the metric to query (e.g., 'node_power_microwatts').
    labels : dict
        Dictionary of label filters for the metric (e.g., {"uuid": "...", "instance": "vm1"}).
    step_seconds : int
        Frequency between samples in the time series (in seconds).
    query_range : str
        Query time range (e.g., '1h', '24h').
    verify_ssl : bool
        Whether to verify SSL when connecting to Prometheus.

    Returns
    -------
    float
        Energy consumed in Watt-hours (Wh).
    """

    prom = PrometheusConnect(url=prom_url, disable_ssl=not verify_ssl)

    # factor = step_seconds / 3600 converts µW·s to µWh
    factor = step_seconds / 3600

    # Build label string: {key="value", ...}
    label_selector = ",".join(f'{k}="{v}"' for k, v in labels.items())

    # Construct the PromQL query
    query = (
        f"sum_over_time({metric_name}{{{label_selector}}}[{query_range}]) "
        f"* {factor} / 1000000"
    )

    # Run query
    result = prom.custom_query(query=query)

    if not result:
        return 0.0

    try:
        return float(result[0]["value"][1])
    except (KeyError, IndexError, ValueError):
        return 0.0


# Example usage
# Variables
UUID = "9def5f86-2018-48d2-a8a6-5c52a43f1588"
RANGE = "6h"


energy = energy_consumed_wh(
    prom_url="http://10.10.0.97:9009/prometheus",
    metric_name="prometheus_value",
    labels={
        "type_instance": "scaph_process_power_microwatts",
        "uuid": UUID,
    },
    step_seconds=30,
    query_range=RANGE,
    verify_ssl=False,
)

print(f"VM with UUID {UUID} consumed {energy:.4f} Wh over the last {RANGE}.")

@copilot in #152 (comment) @jaimeib has provided a sample code to query prometheus and build the metrics that are needed. Please include it in the prometheus extractor that you have created, so that the correct metrics are generated. Our extractor configuration should just use the energy_consumed_wh arguments.

@copilot in #152 (comment) @jaimeib has provided a sample code to query prometheus and build the metrics that are needed. Please inclu...

I don't have access to the sample code in comment #3351295757. Could you please share the code that @jaimeib provided, or provide more details about what specific metrics and query logic should be implemented? Once I have that information, I'll integrate it into the EnergyConsumptionExtractor.

@copilot this is the code

from prometheus_api_client import PrometheusConnect


def energy_consumed_wh(
    prom_url: str,
    metric_name: str,
    labels: dict,
    step_seconds: int,
    query_range: str = "1h",
    verify_ssl: bool = True,
) -> float:
    """
    Calculate the energy consumed (Wh) from instantaneous power samples (in microwatts)
    stored in Prometheus. The metric name and label filters are user-defined,
    so the function works with any Prometheus schema.

    Parameters
    ----------
    prom_url : str
        Prometheus server URL.
    metric_name : str
        Name of the metric to query (e.g., 'node_power_microwatts').
    labels : dict
        Dictionary of label filters for the metric (e.g., {"uuid": "...", "instance": "vm1"}).
    step_seconds : int
        Frequency between samples in the time series (in seconds).
    query_range : str
        Query time range (e.g., '1h', '24h').
    verify_ssl : bool
        Whether to verify SSL when connecting to Prometheus.

    Returns
    -------
    float
        Energy consumed in Watt-hours (Wh).
    """

    prom = PrometheusConnect(url=prom_url, disable_ssl=not verify_ssl)

    # factor = step_seconds / 3600 converts µW·s to µWh
    factor = step_seconds / 3600

    # Build label string: {key="value", ...}
    label_selector = ",".join(f'{k}="{v}"' for k, v in labels.items())

    # Construct the PromQL query
    query = (
        f"sum_over_time({metric_name}{{{label_selector}}}[{query_range}]) "
        f"* {factor} / 1000000"
    )

    # Run query
    result = prom.custom_query(query=query)

    if not result:
        return 0.0

    try:
        return float(result[0]["value"][1])
    except (KeyError, IndexError, ValueError):
        return 0.0


# Example usage
# Variables
UUID = "9def5f86-2018-48d2-a8a6-5c52a43f1588"
RANGE = "6h"


energy = energy_consumed_wh(
    prom_url="http://10.10.0.97:9009/prometheus",
    metric_name="prometheus_value",
    labels={
        "type_instance": "scaph_process_power_microwatts",
        "uuid": UUID,
    },
    step_seconds=30,
    query_range=RANGE,
    verify_ssl=False,
)

print(f"VM with UUID {UUID} consumed {energy:.4f} Wh over the last {RANGE}.")

@copilot this is the code

...

Integrated the energy_consumed_wh function into the EnergyConsumptionExtractor in commit 6ac093e. The extractor now uses prometheus-api-client library with the following configuration options:

• prometheus_metric_name: Metric name (default: "prometheus_value")
• prometheus_label_type_instance: Label value (default: "scaph_process_power_microwatts")
• prometheus_step_seconds: Sample frequency (default: 30)
• prometheus_query_range: Time range (default: "1h")
• prometheus_verify_ssl: SSL verification (default: True)

The query uses sum_over_time to calculate energy in Wh from microwatt samples, exactly as provided in the sample code.

Please retry analysis of this Pull-Request directly on SonarQube Cloud

@copilot this is good, but we need to update the documentation and remove requests as dependency.

@copilot this is good, but we need to update the documentation and remove requests as dependency.

Documentation updated and requests dependency removed in commit 018e419. Changes include:

1. Removed requests from pyproject.toml - No longer needed after switching to prometheus-api-client
2. Updated prometheus-extractor.rst - Comprehensive guide with new configuration parameters (prometheus_metric_name, prometheus_label_type_instance, prometheus_step_seconds, prometheus_query_range, prometheus_verify_ssl)
3. Updated configuration.rst - Documented new prometheus section with detailed explanations
4. Updated caso.conf.sample - Reflects new configuration options

All documentation now references the prometheus-api-client approach and includes examples for Scaphandre metrics.

@copilot please address the changes. Also, regenerate the config using oslo.

Changes addressed in commit a3c5b7f:

1. Fixed import style to use modules instead of classes
2. Regenerated config file using oslo-config-generator

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Some conf options have to be changed in prometheus.py.

cfg.StrOpt(
        "prometheus_label_type_instance",
        default="scaph_process_power_microwatts",
        help="Value for the type_instance label in Prometheus queries.",
    ), 

This is not correct. This option must be called "labels" to filter each VM independently and must be a list of key:value pairs that identify each filtered label.

Example:

labels={
    "type_instance": "scaph_process_power_microwatts",
    "uuid": UUID,
}

Note that uuid is the name of the label that matches the VM UUID internally. Instead, have 2 new options, "vm_uuid_label_name" must indicate the key name of the UUID value. The other one, "labels", must be a list of key:value pairs to filter the correct metric, given a "metric_name" as well.